COMMENT ⊗   VALID 00010 PAGES
C REC  PAGE   DESCRIPTION
C00001 00001
C00002 00002	.SPACING 1
C00003 00003	.COMPACT INDENT 5
C00012 00004	WHY ASCRIBE MENTAL QUALITIES?
C00022 00005	TWO METHODS OF DEFINITION
C00043 00006	EXAMPLES OF SYSTEMS WITH MENTAL QUALITIES
C00076 00007	"GLOSSARY" OF MENTAL QUALITIES
C00091 00008	OTHER VIEWS ABOUT MIND
C00092 00009	.PORTION NOTES
C00101 00010	REFERENCES
C00102 ENDMK
C⊗;
.SPACING 1;
.TURN ON "{";
.DEVICE XGP;
.FONT 1 "BASL30"; TURN ON "%";
.FONT 2 "BASI30";
.FONT 3 "basb30";
.FONT 4 "SUB";
.FONT 5 "SUP";
.at "⊗"word" "	⊂ ("%2word%* ") ⊃;
.COUNT ITEM
.COUNT SUBITEM IN ITEM
.COUNT NOTE inline
.AT "~" TEXT "~"; ⊂
.SEND NOTES ⊂
{NOTE!}. TEXT

.⊃;
.⊃;
.AT "$" ⊂ NEXT NOTE; ("%5"&NOTE!&"%*");⊃;
.AT "#" ⊂NEXT ITEM;(ITEM!);⊃;
.AT "&" ⊂NEXT SUBITEM;ONCE INDENT 10;(ITEM!&"."&SUBITEM!);⊃;
.PAGE FRAME 53 HIGH 80 WIDE;
.SECNUM←0
.MACRO NS(A)  ⊂ SECNUM←SECNUM+1
.SKIP TO COLUMN 1
.ONCE CENTER;
%3{SECNUM}.  A%*
.  ⊃
.EVERY HEADING(PARTIAL DRAFT, ,{DATE})
.EVERY FOOTING(,{PAGE},)
.COMPACT; INDENT 5;
.NEXT PAGE
.PLACE TEXT
.SELECT 1
.ONCE CENTER
(this draft compiled at {TIME} on {DATE})
.SKIP 2
.ONCE CENTER
ASCRIBING MENTAL QUALITIES TO MACHINES


	To ascribe certain %2beliefs%1, %2knowledge%1, %2free will%1,
%2intentions%1,  %2consciousness%1,  %2abilities%1  or %2wants%1 to a
machine or computer program is %3legitimate%1  whenever  such  ascription
expresses  the  same  information  about  the machine that it expresses
about a person.     It  is  %3useful%1  when  the  ascription  helps  us
understand the structure of the machine, its past or future behavior,
or how to repair or improve it.
It is perhaps never %3logically required%1 even for humans,
but a practical theory of the behavior of machines
or humans may require mental qualities or qualities isomorphic to them.
Theories of belief,  knowledge  and
wanting can be constructed for machines in a simpler setting than for
humans and later applied to humans.    Ascription of mental qualities
is %3most straightforward%1 for machines of known structure such as
thermostats and computer operating systems, but is %3most useful%1  when
applied  to  machines  whose  structure  is  very incompletely known.

	The above  views come from  work in  artificial intelligence$
(abbreviated  AI).  They can  be taken as asserting that
many of the philosophical problems  of mind take a practical form  as
soon  as one  takes  seriously  the idea  of  making machines  behave
intelligently.  In particular, AI raises for machines two issues that
have heretofore been considered only in connection with people.

	First, in designing intelligent programs  and looking at them
from  the outside  we need  to determine  the conditions  under which
specific mental and volitional terms  are applicable.  We can  exemplify
these  problems by  asking  when  it is  legitimate  to  say about  a
machine, %2" It knows I want a reservation to Boston, and it can give
it to me, but it won't"%1.

~We don't claim that the work in artificial intelligence has yet shown
how to reach human-level intellectual performance.
Our approach to the AI problem involves identifying the intellectual
mechanisms required for problem solving and describing them precisely,
and therefore we are at the end of the philosophical
spectrum that requires everything to be formalized in mathematical logic.
It  is sometimes  said  that  one
studies philosophy  in order to advance beyond  one's untutored naive
world-view, but unfortunately for  artificial intelligence, no-one has
yet been able to give a precise description of even a naive world-view.
~
	Second, when we  want a %3generally intelligent%1$ computer
program, we must build into it a %3general view%1 of what the world
is like with  especial attention to facts  about how the  information
required to solve problems is to be  obtained and used.  Thus we must
provide   it   with   some   kind   of   %2metaphysics%1 (general
world-view)  and %2epistemology%1 (theory of knowledge) however naive.

~Present AI programs operate in limited domains, e.g. play particular
games, prove theorems in a particular logical system, or understand
natural language sentences covering a particular subject matter and
with other semantic restrictions.  General intelligence will require
general models of situations changing in time, actors with goals
and strategies for achieving them, and knowledge about how information
can be obtained.~

	As  much  as  possible,  we  will  ascribe  mental  qualities
separately from  each other instead of bundling  them  in a
concept of mind.   This is necessary,  because present machines  have
rather   varied  little   minds;  the   mental  qualities   that  can
legitimately  be ascribed to them are few  and differ from machine to
machine.   We  will  not  even try  to  meet  objections  like,
%2"Unless it also  does X, it is illegitimate to  speak of its having
mental qualities."%1

	Machines  as  simple  as thermostats  can  be  said  to  have
beliefs,  and having  beliefs seems  to be  a characteristic  of most
machines  capable of  problem  solving  performance.    However,  the
machines mankind has so far found  it useful to construct rarely have
beliefs  about beliefs.    (Beliefs about  beliefs will be  needed by
computer programs to reason about what  knowledge they
lack and where  to get it).  Other mental qualities,  such as love and
hate, seem peculiar  to human-like motivational  structures and  will
not  be required  for  intelligent behavior,  but  we could  probably
program computers to exhibit them if we wanted to, because our common
sense notions about them  translate readily into certain program  and
data  structures.   Still  other  mental  qualities, e.g.  humor  and
appreciation  of beauty,  seem much  harder to  model.  While  we will be
quite liberal  in  ascribing ⊗some  mental  qualities even  to  rather
primitive machines, we will try to be conservative in our criteria
for ascribing any ⊗particular quality.

	The successive sections of this paper will give philosophical
and AI reasons for ascribing beliefs to machines, two new forms of
definition that seem necessary for defining mental qualities and
examples of their use, examples of systems to which mental
qualities are ascribed, some first attempts at defining a variety
of mental qualities, some criticisms of other views on mental
qualities, notes, and references.

.SKIP TO COLUMN 1
.ONCE CENTER
WHY ASCRIBE MENTAL QUALITIES?


	%3Why  should we want to  ascribe beliefs to  machines  at all?%1
This is the opposite question to that involved in the controversy over
%2reductionism%1.  Instead of asking how mental qualities can be
%3reduced%1 to physical ones, we ask how to %3ascribe%1 mental qualities
to physical systems.  In our opinion, this question is more natural
and may lead to better answers to the questions of reductionism.

	To put  the issue  sharply, consider  a computer program  for
which  we posess complete listings.   The behavior of  the program in
any environment is determined from  the structure of the program  and
can be  found out  by simulating  the action of  the program  and the
environment  without  having  to deal  with  any  concept  of belief.
Nevertheless, there  are  several reasons  for  ascribing belief  and
other mental qualities:

.ITEM←0;
	#. Although we may know the program, its state at a given moment
is usually not directly observable, and the conclusions we can draw
about its current state may be more readily expressed by ascribing
certain beliefs or wants than in any other way.

	#. Even if we can simulate the interaction of our program
with its environment using another more comprehensive program, the
simulation may be a billion times too slow.  We also
may not have the initial conditions of the environment or its laws of
motion in  a suitable  form, whereas  it may  be feasible  to make  a
prediction of  the effects of the  beliefs we ascribe to  the program
without any computer at all.

	#. Ascribing beliefs may allow deriving general statements
aobut the program's behavior that could not be obtained from any
finite number of simulations.

	#.   The belief and goal structures we ascribe to the program
may be easier to understand than the details of  program as expressed
in its listing.

	#. The belief and goal structure is likely to be close to the
structure the designer  of the  program had in  mind, and  it may  be
easier to debug the program in terms  of this structure than directly
from  the listing.   In  fact, it  is often  possible for  someone to
correct a fault by reasoning  in general terms about the  information
in a program or machine, diagnosing  what is wrong as a false belief,
and   looking  at  the  details  of   the  program  or  machine  only
sufficiently to determine how  the false belief is represented  and
what mechanism caused it to arise.$

	All the above reasons for ascribing beliefs are epistemological.
i.e. ascribing beliefs is needed to adapt to limitations on our
ability to acquire knowledge, use it for prediction, and establish
generalizations in terms of the elementary structure of the program.
Perhaps this is the general reason for ascribing higher levels of
organization to systems.

~	This  kind  of  teleological  analysis  is  often  useful  in
understanding natural organisms as well as machines.  Here  evolution
takes  the place  of  design and  we  often understand  the  function
performed by  an organ before we  understand its detailed physiology.
Teleological analysis  is  applicable to  psychological  and  social
phenomena in  so far as  these are designed  or have been  subject to
selection.

	However,  teleological analysis fails when applied to aspects
of nature which  have neither been  designed nor produced by  natural
selection from a population.   Much medieval science was based on the
Judeo-Christian-Moslem religious hypothesis that  the details of  the
world were designed by God  for the benefit of man.   The strong form
of  this hypothesis was abandoned  at the time of  Galileo and Newton
but occasionally recurs.   Barry Commoner's (1972) axiom of  ecology
"Nature knows best"  seems to be mistakenly based on  the notion that
nature  as a  whole  is the  result of  an evolutionary  process that
selected the "best nature".~

	Computers give rise to numerous examples of building a higher
structure on the basis of a lower and conducting subsequent analyses
using the higher structure.  The geometry of the electric fields in
a transistor and its chemical composition give rise to its properties
as an electric circuit element.
Transistors are combined in small circuits and powered in standard
ways to make logical elements such as ANDs, ORs, NOTs and flip-flops.
Computers are designed with these logical elements
to obey a desired order code; the designer usually needn't
consider the properties of the transistors as circuit elements.
The designer of a higher level language works with the order
code and doesn't have to know abut the logic; the user of
the higher order language needn't know the computer's order code.

	In the above cases, users of the  higher level can completely
ignore  the lower level, because the  behavior of the higher  level system is
completely determined by  the values of  the higher level  variables;
e.g.   in order to determine  the outcome of a  computer program, one
should  not have to look at the  states of flip-flops.  However, when
we ascribe mental structure to humans or goals to  society, we always
get highly  incomplete systems; the  higher level behavior  cannot be
fully predicted  from  higher  level observations  and  higher  level
"laws" even when the underlying lower level behavior is determinate.

	Besides the above philosophical reasons for ascribing mental
qualities to machines, I shall argue that in order to make machines
behave intelligently, we will have to program them to ascribe beliefs
etc. to each other and to people.

→→→→Here there will be more on machine's models of each others minds.←←←←


.SKIP TO COLUMN 1
.ONCE CENTER
TWO METHODS OF DEFINITION
.ONCE CENTER
AND THEIR APPLICATION TO MENTAL QUALITIES

	In our opinion, a major source of problems in defining mental
and other philosophical concepts is the weakness of the methods of
definition that have been %2explicitly%1 used.  We will consider two
kinds of definition: %2second order structural definition%1 and
%2definition relative to an approximate theory%1 and their application
to defining mental qualities.

.ITEM←0;

#. %3Second Order Structural Definition.%1

	Structural definitions of qualities are given in terms of the
state of the  system being described while behavioral definitions are
given in terms of actual or potential behavior$.

	~Behavioral  definitions are often  favored in  philosophy.  A
system is  defined to  have  a certain  quality if  it behaves  in  a
certain way  or  is  %2disposed%1 to behave  in a  certain way.
Their  ostensible  virtue  is  conservatism;   they  don't  postulate
internal  states that  are unobservable  to  present science  and may
remain unobservable.
However, such  definitions are awkward for  mental qualities, because,
as common sense  suggests, a  mental quality  may  not  result  in
behavior, because another mental  quality may prevent it; e.g.  I may
think you are  thick-headed, but politeness may prevent my saying so.
Particular  difficulties  can  be  overcome,  but  an  impression  of
vagueness remains.  The liking  for behavioral definitions stems from
caution, but  I would interpret scientific experience as showing that
boldness in postulating  complex structures of unobserved  entities -
provided it is  accompanied by a willingness to  take back mistakes -
is more likely to be rewarded by understanding of  and  control  over
nature   than   is   positivistic   timidity.    It  is  particularly
instructive to imagine a determined behaviorist trying to figure  out
an  electronic  computer.  Trying to define each quality behaviorally
would get him nowhere;  only  simultaneously  postulating  a  complex
structure  including  memory, arithmetic unit, control structure, and
input-output would yield predictions  that  could  be  compared  with
experiment.~

	In the case of a specific known machine, one can often give a
%2first  order  structural  definition%1.    Thus  we  might  give  a
predicate %2B(s,p)%1 such that if the  machine is in state %2s%1,  it
is said to  believe the sentence  %2p%1 provided %2B(s,p)%1  is true.
However, I don't think there is a general definition of belief having
this form that applies to all machines in all environments.

	Therefore we give a %2second order predicate%1  β%2(B,W)%1 that
tells whether we regard the  first order predicate %2B(s,p)%1 as a
"good" notion  of belief in the %2world W%1.  Such a predicate β will
be  called a  %2second  order  definition%1; it  gives  criteria  for
criticizing   an  ascription  of   a  quality   to  a  system.
Axiomatizations of  belief  give   rise  to  second   order
definitions, and we suggest that both our  common sense and scientific
usage  of not-directly-observable qualities  corresponds more closely
to second  order structural  definition than  to any kind of behavioral
definition.   It  should  be  noted that  a  second order  definition
cannot  guarantee  that  there exist  predicates  %2B%1  meeting the
criterion β or that such a %2B%1 is unique.   It can  also  turn  out
that  a quality is best defined as a member of a group of related
qualities.

	The  second   order   definition  criticizes   whole   belief
structures rather than  individual beliefs.  We can  treat individual
beliefs  by  saying  that a  system  believes  %2p%1  in state  %2s%1
provided all "reasonably good"  %2B%1's satisfy %2B(s,p)%1.  Thus  we
are distinguishing the "intersection" of the reasonably good %2B%1's.

	It  seems  to  me  that  there should  be  a  metatheorem  of
mathematical  logic asserting  that not all  second order definitions
can  be reduced  to  first  order definitions  and  further  theorems
characterizing  those  second   order  definitions  that  admit  such
reductions.  Such  technical results, if  they can be  found, may  be
helpful in  philosophy and in  the construction of  formal scientific
theories.  I would conjecture that many of the informal philosophical
arguments that certain mental  concepts cannot be reduced to  physics
will turn out to be sketches of arguments that these concepts require
second (or higher) order definitions.


	Here  is a deliberately imprecise second order definition of
belief.  For each state %2s%1 of the  machine and
each  sentence %2p%1 in  a suitable  language %2L%1,
a belief predicate %2B(s,p)%1 assigns  truth  or falsity
according to whether the machine is considered to believe %2p%1  when
it  is  in state  %2s%1.    The  language  %2L%1 is  chosen  for  our
convenience, and  there is no assumption  that the machine explicitly
represents sentences of %2L%1 in any way.  Thus we can talk about the
beliefs of Chinese, dogs, thermostats, and computer operating systems
without  assuming that they  use English or our  favorite first order
language.

	We now subject %2B(s,p)%1 to the certain criteria; i.e. β⊗(B,W) 
is considered true provided the following conditions are satisfied:

	&. The set %2Bel(s)%1 of beliefs, i.e. the set of %2p%1's for
which  %2B(s,p)%1 is  assigned true  contains  sufficiently "obvious"
consequences of some of its members.

	&.   %2Bel(s)%1 changes in a  reasonable  way when the  state
changes.    We   like  new  beliefs  to  be  logical  or  "plausible"
consequences of old ones or to come in as %2communications%1  in some
language on the input lines or  to be %2observations%1, i.e.  beliefs
about the environment the information for which comes in on the input
lines.  The set of beliefs should not change too rapidly as the state
changes.

	&.   We prefer  the set  of beliefs  to be  as consistent  as
possible.  (Admittedly, consistency  is not a quantitative concept in
mathematical logic -  a system is  either consistent or  not, but  it
would seem that we  will sometimes have to ascribe  inconsistent sets
of beliefs to machines and people.  Our intuition says that we should
be able to maintain areas of  consistency in our beliefs and that  it
may be  especially important  to avoid  inconsistencies in  the machine's
purely analytic beliefs).

	&.  Our criteria for belief systems can be strengthened if we
identify some of the  machine's beliefs as expressing goals,  i.e. if
we have beliefs  of the form "It would be good if  ...".  Then we can
ask that the machine's behavior  be somewhat %2rational%1, i.e.  %2it
does what  it  believes will  achieve its  goals%1. The  more of  its
behavior  we can  account for in  this way,  the better we  will like
%2B(s,p)%1.  We also would like to account for internal state changes
as changes in belief in so far as this is reasonable.

	&.  If the machine communicates, i.e. emits sentences in some
language  that  can  be  interpreted  as  assertions,  questions  and
commands, we will want the assertions to be among  its beliefs unless
we are  ascribing to it  a goal or  subgoal that involves  lying.  In
general, its  communications  should  be such  as  it  believes  will
achieve its goals.

	&.  Sometimes we shall want to ascribe introspective beliefs,
e.g. a belief that it does not know how to fly to Boston or even that
it doesn't know what it wants in a certain situation.

	&. Finally, we will prefer a more economical ascription %2B%1
to a less economical one.  The  fewer beliefs we ascribe and the less
they change with state consistent with accounting for he
behavior and the internal state changes, the better we will like it.

	The above  criteria have  been  formulated somewhat  vaguely.
This  would be bad if there  were widely different ascriptions of  beliefs
to a particular machine that all met our  criteria or if the criteria
allowed  ascriptions that  differed widely from  our intuitions.   My
present opinion is that more thought will make the  criteria somewhat
more precise  at no cost  in applicability, but that  they %2should%1
still remain  rather vague, i.e. we shall want to ascribe belief in a
%2family%1 of cases.
However, even at the present level of vagueness, there probably
won't be radically different equally "good" ascriptions of belief for
systems of practical interest.  If there were, we would notice the
ambiguity in ordinary language.

	Of course we will need precise axiomatizations of belief and
other mental qualities to build into particular intelligent computer programs.


#. %3Definitions relative to an approximate theory%1.


	Certain  concepts,  e.g. %2X can do Y%1, are meaningful only in
connection with a rather complex theory.   For  example,  suppose  we
denote the state of the world by %2s%1, and suppose we have functions
%2f%41%2(s)%1,...,%2f%4n%2(s)%1  that  are  directly  or   indirectly
observable.  Suppose further that %2F(s)%1 is another function of the
world-state  but  that  we   can   approximate   it by

	%2F"(s) = F'(f%41%2(s),...,f%4n%2(s))%1.

	Now  consider  the counterfactual
conditional sentence, "If %2f%42%2(s)%1 were 4, then %2F(s)%1 would be
3 - calling the present state of the world %2s%40%1." By itself, this
sentence has no meaning, because no definite state %2s%1 of the world
is  specified  by  the  condition.   However, in the framework of the
functions %2f%41%2(s),...,f%4n%2(s)%1 and the given approximation  to
%2F(s)%1,  the  assertion  can be verified by computing ⊗F' with all
arguments except the second having the  values  associated  with  the
state %2s%40%1 of the world.

	This gives rise to some remarks:

&. The most straightforward case of counterfactuals arises when
the state of a phenomenon has a distinguished Cartesian product
structure.  Then the meaning of a change of one component without
changing the others is quite clear.  Changes of more than one
component also have definite meanings.  This is a stronger
structure than the %2possible worlds%1 structure discussed in
(Lewis 1973).

&. The usual case is one in which the state %2s%1 is a substantially
unknown entity and the form of the function %2F%1 is also
unknown, but the values of %2f%41%2(s),...,f%4n%2(s)%1 and
the function %2F'%1 are much better known.
Suppose further that %2F"(s)%1 is known to be only a fair approximation
to %2F(s)%1.  We now have a situation in which the counterfactual
conditional statement is meaningful as long as it is not examined too
closely, i.e. as long as we are thinking of the world in terms of
the values of %2f%41%2,...,f%4n%1, but when we go beyond the
approximate theory, the whole meaning of the sentence seems to
disintegrate.

	Our idea is that this is a very common phenomenon. In
particular it applies to statements of the form %2"X can do Y"%1.
Such statements can be given a precise meaning in terms
of a system of interacting automata as is discussed in detail in
(McCarthy and Hayes 1970).  We say that Automaton 1 can put Automaton
3 in state 5 at time 10 by asking a question about an automaton system
in which the outputs from Automaton 1 are replaced by inputs from
outside the system.  Namely, we ask whether there is a sequence of
inputs to the new system that ⊗would put Automaton 3 in state 5 at time 10;
if yes, we say that Automaton 1 ⊗could do it in the original system
even though we may be able to show that he won't emit the necessary
outputs.  In that paper, we argue that this definition corresponds
to the intuitive notion of %2X can do Y.%1.

	What was not noted in that paper is that modelling the
situation by the particular system of interacting automata is
an approximation, and the sentences involving
⊗can derived from the approximation
cannot necessarily be translated into single assertions about the
real world. 

	I contend  that the statement,  %2"I can go  skiing tomorrow,
but I don't intend to, because I want to finish this paper"%1 has the
following properties:

	1. It has a precise meaning in a certain approximate theory of
in which I and my environment are considered as collections of interacting
automata.

	2. It cannot be directly interpreted as a statement about the world
itself, because it can't be stated in what total configurations of the
world the success of my attempt to go skiing is to be validated.

	3. The approximate theory within which the statement is meaningful
may have an objectively preferred status in that it may be the only theory
not enormously more complex that enables my actions and mental states to
be predicted.

	4. The statement may convey useful information.

Our conclusion is that the statement is %3true%1, but in a sense that
depends essentially on the approximate theory, and that this intellectual
situation is normal and should be accepted.  We further conclude that
the old-fashioned common sense analysis of a personality into %2will%1
and %2intellect%1 and other components may be valid and might be put
on a precise scientific footing using %2definitions relative to an
approximate theory%1.


.SKIP TO COLUMN 1
.ONCE CENTER
EXAMPLES OF SYSTEMS WITH MENTAL QUALITIES

	Let us  consider some  examples of machines  and programs  to
which we may ascribe belief and goal structures.

.ITEM←0;
	#.  %3Thermostats.%1   Ascribing beliefs to simple thermostats is
not really necessary, because their operation can be well  understood
without it.  However, their very simplicity  makes it clearer what is
involved  in the  ascription, and  we maintain (to  some extent  as a
provocation aimed  at  those who  regard  attribution of  beliefs  to
machines  as mere  intellectual  sloppiness) that  the ascription  is
legitimate even if unnecessary.$

~Whether a system has beliefs and other mental qualities is not
primarily a matter of complexity of the system.  Although cars are
more complex than thermostats, it is hard to ascribe beliefs or
goals to them, and the same is perhaps true of the basic hardware of a
computer, i.e. the part of the computer that executes the program
without the program itself.
~
	First  let us consider a simple  thermostat that turns off the
heat when the  temperature is a degree  above the temperature set  on
the thermostat,  turns on the heat  when the temperature  is a degree
below the desired  temperature, and leaves  the heat as  is when  the
temperature  is   in  the  two   degree  range  around   the  desired
temperature. The simplest belief predicate %2B(s,p)%1 ascribes belief
to only two sentences:  "The room is too cold"  and "The room is  too
hot", and these  beliefs are assigned to states of  the thermostat so
that  in  the  two  degree range,  neither  is  believed.    When the
thermostat believes  the  room is  too cold  or too  hot  it sends  a
message to that effect to the furnace. A slightly more complex belief
predicate could also  be used in  which the thermostat  has a  belief
about what the temperature should be and another belief about what it
is.   It is not clear  which is better, but if  we wished to consider
possible errors in the thermometer, the then we would have to be able
to ascribe beliefs  about what the temperature is. We  do not ascribe
to  it any other  beliefs; it has  no opinion even  about whether the
heat is on or off or about the weather or about who won the battle of
Waterloo.    Moreover, it  has  no introspective  beliefs,  i.e.   it
doesn't believe that it believes the room is too hot.

	The temperature control system in  my house may be  described
as  follows: Thermostats  upstairs and  downstairs  tell the  central
system  to turn  on or shut  off hot  water flow  to these areas.   A
central water-temperature thermostat tells the furnace to turn  on or
off  thus  keeping the  central  hot  water  reservoir at  the  right
temperture.  Recently it was too hot upstairs, and the question arose
as to whether the upstairs thermostat mistakenly  %2believed%1 it was
too  cold  upstairs  or  whether  the furnace  thermostat  mistakenly
%2believed %1 the  water was too  cold.  It  turned out that  neither
mistake was made; the downstairs controller %2tried%1 to turn off the
flow  of water but  %2couldn't%1, because the  valve was  stuck.  The
plumber came  once  and found  the trouble,  and  came again  ten  a
replacement valve  was ordered.   Since the services of  plumbers are
increasingly  expensive, and  microcomputers are  increasingly cheap,
one is led to design a temperature control system that would %2know%1
a lot more about the thermal  state of the house and its own state of
health.

	In the first  place, while the  system %2couldn't%1 turn  off
the flow of hot water  upstairs, there is no reason  to ascribe to
it the  %2knowledge%1 that it couldn't, and  %2a fortiori%1 it had no
ability to %2communicate%1 this %2fact%1  or to take it into  account
in controlling the system.  A more advanced system would know whether
the  %2actions%1 it %2attempted%1 succeeded, and it would communicate
failures and adapt  to them.  (We  adapted to the failure  by turning
off  the whole  system  until the  whole  house cooled  off and  then
letting the two parts warm up  together.  The present system has  the
%2physical  capability%1  of  doing  this  even   if  it  hasn't  the
%2knowledge%1 or the %2will%1.


	#. %3Self-reproducing  intelligent  configurations  in  a  cellular
automaton world.%1  A ⊗cellular ⊗automaton system assigns to each vertex
in  a  certain graph a finite automaton.  The state of each automaton
at time ⊗t+1 depends on its state at time %2t%1 and the states  of
its  neighbors  at  time %2t%1. The most common graph is the array of
points ⊗(x,y) in the plane with integer co-ordinates ⊗x and ⊗y.   The
first  use of cellular automata was by von Neumann (196?) who found a
27 state automaton that could be used to  construct  self-reproducing
configuration   that  were  also  universal  computers.    The  basic
automaton in von Neumann's system had a distinguished state called  0
and  a  point in state 0 whose four neighbors were also in that state
would remain in state 0.  The initial configurations  considered  had
all  but  a  finite  number of cells in state 0, and, of course, this
property would persist although the number of  non-zero  cells  might
grow indefinitely with time.

	The  self-reproducing  system used the states of a long strip
of non-zero cells as a "tape" containing instructions to a "universal
constructor"  configuration  that  would  construct  a  copy  of  the
configuration to be reproduced but with each cell in a passive  state
that  would  persist  as  long  as its neighbors were also in passive
states.  After the construction phase, the tape would  be  copied  to
make  the  tape for the new machine, and then the new system would be
set in motion by activating one of its cells. The  new  system  would
then move away from its mother and the process would start over.  The
purpose of the design was to  demonstrate  that  arbitrarily  complex
configurations  could  be  self-reproducing  -  the  complexity being
assured by also requiring that they be universal computers.

	Since von Neumann's time, simpler basic cells admitting
self-reproducing universal computers have been discovered.  Perhaps
the simplest is the two state Life automaton of John Conway (196?).
The state of a cell at time ⊗t+1 is determined its state at time ⊗t 
and the states of its eight neighbors at time ⊗t.  Namely, if
a point whose state is 0 will change to state 1 if exactly three
of its neighbors are in state 1.  A point whose state is 1 will
remain in state 1 if two or three of its neighbors are in state 1.
In all other cases the state becomes or remains 0.

	Conway's initial intent was to model a birth and death
process whereby a cell is born (goes into state 1) if it has the
right number of living neighbors (namely three) and dies if it
is either too lonely (has none or one neighbor in state 1) or is
overcrowded (has four or more living neighbors).  He also asked
whether infinitely growing configurations were possible, and
Gosper first proved that there were.
More than that, it was shown that self-reproducing universal
computers could be built up as Life configurations.

	Let us now imagine that there are a number of such
self-reproducing universal computers operating in the Life
plane and suppose that they have been programmed to study
the properties of their world and to communicate among themselves
about it pursuing various goals co-operatively and competitively.
Let's call these configurations robots.
In some respects their intellectual and scientific problems
will be like ours, but in one major respect they will live
in a simpler world than ours has been shown to be.  Namely, 
the fundamental physics of their world is that of the life
automaton, and there is no obstacle to each robot ⊗knowing 
this physics at least in the sense of being able to simulate
the evolution of a life configuration starting in the initial
state.  Moreover, if the initial state of the robot world is finite it can
have been recorded in each robot in the beginning or else
recorded on a strip of cells that the robots can read.
(The infinite regress of having to describe the description
is avoided by a convention that the description is not
described, but can be read ⊗both as a description of the
world ⊗and as a description of the description itself.)

	These robots then know the initial state of their world
and its laws of motion.  Therefore, they can simulate as much
of their world's history as they want assuming that each of them
can grow into unoccupied space so as to have memory to store
the states of the world being simulated.  Of course, this simulation
is much slower than real time so they can never catch up with the
present let alone predict the future.  This is quite evident if we
imagine the simulation carried out in a straightforward way in which
a list of currently active cells in the simulated world is updated
according to the Life rule, but it also applies to clever mathematical
methods that might predict millions of steps ahead.  (Some Life
configurations, e.g. static ones or ones containing single ⊗gliders 
or ⊗cannon can have their distant futures predicted with little
computing.)  Namely, if there were an algorithm for such prediction,
a robot could be made that would predict its own future and then
disobey the prediction.

	Now we come to the point of this long disquisition.  Suppose
that we wish to program a robot to be successful in the Life world in
competition or co-operation with the others.  Without any idea
of how to give a mathematical proof, I will claim that our robot
will need programs that ascribe purposes and beliefs to its fellow robots
and  predict how they will react to our robot's own
actions by assuming that %2they will act in ways that they believe
will achieve their goals%1.  Our robot can acquire these mental theories
in several ways:  First, we might design in such programs and
install them in the initial configuration of the world.  Second, it might
be programmed to acquire these programs by induction from its experience
and perhaps pass them on to others through an educational system.  Third,
it might derive the psychological laws from the fundamental physics of
the world and its knowledge of the initial configuration
or it might discover how robots are built from Life cells by doing
experimental "biology".

	Knowing the Life  physics without some information  about the
initial configuration is insufficient to derive the %2psychological%1
laws, because  robots can  be constructed  in the  Life  world in  an
infinity of ways.  This follows from the "folk theorem" that the Life
automaton  is universal in the sense  that any cellular automaton can
be constructed by taking sufficiently large squares of  Life cells as
the basic cell of the other automaton.$

	Our own intellectual position is more difficult than that of
the Life robots.  We don't know the fundamental physics of our world,
and we can't even be sure that its fundamental physics is describable
in finite terms.  Even if we knew the physical laws, they seem to
preclude precise knowledge of an initial state and precise calculation
of its future both for quantum mechanical reasons and because the
continuous functions involved can't necessarily be described by
a finite amount of information.

~	Our ability to derive the laws of
higher levels of organization from knowledge of lower level laws is
also limited by universality.  Namely, while there appears to be
essentially one possible chemistry allowed by the laws of physics,
the laws of physics and chemistry allow many biologies, and, because
the neuron is a universal computing element, an arbitrary mental
structure is allowed by basic neurophysiology.  This means that in
order to determine human mental structure one must either make
psychological experiments or determine the actual anatomical structure
of the brain and the information stored in it or reason from the
fact that the brain is capable of certain problem solving performance
to the structures that must be present to provide that performance.
In this respect, our position is similar to that of the Life robot.~

	The point of the cellular automaton robot example is that much
of human mental structure is not an accident of evolution or even
of the physics of our world, but is required for successful problem
solving behavior and must be designed into or evolved by any system
that exhibits such behavior.


	#. %3Computer time-sharing systems.%1  These complicated
computer programs allocate computer time and other resources among users.
They allow each user of the computer to behave as though he had
a computer of his own, but also allow them to share files of
data and programs and to communicate with each other.
They are often used for many years with continual small changes, and
and the people making the changes and correcting errors
are often different not the original authors of the system.
A person confronted with the task of correcting a malfunction or
making a change in a time-sharing system can conveniently use a
mentalistic model of the system.

	Thus suppose a user complains that the system
will not run his program.  Perhaps the system believes that he
doesn't want to run, perhaps it persistently believes that he
has just run, perhaps it believes that his quota of computer
resources is exhausted, or perhaps it believes that his program
requires a resource that is unavailable.  Testing these hypotheses
can often be done with surprisingly little understanding of the
internal workings of the program.

→→→→→There will be more examples here of the belief of time-sharing systems.←←←

	#. %3Programs designed to reason.%1  Suppose we explicitly design a
program to represent information by sentences in a certain language
stored in the memory of the computer and decide what to do by making
inferences, and doing what it concludes will advance its goals.  Naturally,
we would hope that our previous second order definition of belief will
"approve of" a %2B(p,s)%1 that ascribed to the program believing the
sentences explicitly built in.  We would be somewhat embarassed if
someone were to show that our second order definition approved as
well or better of an entirely different set of beliefs.

	Such a program was first proposed in (McCarthy 1960), and here is how
it might work:

	Information about the world is stored in a wide variety of
data structures.  For example, a visual scene received by a TV
camera may be represented by a 512x512x3 array of numbers representing
the intensities of three colors at the points of the visual field.
At another level, the same scene may be represented by a list of regions,
and at a further level there may be a list of physical objects and their
parts together with other information about these objects obtained from
non-visual sources.  Moreover, information about how to solve various
kinds of problems may be represented by programs in some programming
language.

	However, all the above representations are subordinate to
a collection of sentences in a suitable first order language that
includes set theory.  By subordinate, we mean that there are sentences
that tell what the data structures represent and what the programs do.
New sentences can arise by a variety of processes: inference from
sentences already present, by computation from the data structures representing
observations, ...

→→→→→There will be more here about what mental qualities should be programmed.←←←

.SKIP TO COLUMN 1
.ONCE CENTER
"GLOSSARY" OF MENTAL QUALITIES

	In this section we give short "definitions" for machines of a collection
of mental qualities.  We include a number of terms which give us
difficulty with an indication of what the difficulties seem to be.

.ITEM←0;

	#. %3Actions%1.  We want to distinguish the actions of a being
from events that occur in its body and that affect the outside
world.  For example, we wish to distinguish a random twitch
from a purposeful movement.  This is not difficult %2relative
to a theory of belief that includes intentions%1.  One's purposeful
actions are those that would have been different had one's intentions
been different.  This requires that the theory of belief have sufficient
Cartesian product structure so that the counterfactual conditional
`"if its intentions had been different" is defined in the theory.
As explained in the section on definitions relative to an approximate
theory, it is not necessary that the counterfactual be given a
meaning in terms of the real world.

	#. %3Introspection and self-knowledge.%1

	We  say that  a  machine introspects  when it  comes  to have
beliefs about its own mental  state.  A simple form of  introspection
takes  place  when  a  program  determines  whether  it  has  certain
information  and if not asks for it.   Often an operating system will
compute a check  sum of itself  every few minutes  to verify that  it
hasn't been changed by a software or hardware malfunction.

	In principle,  introspection is easier  for computer programs
than for people, because the entire memory in which programs and data
are stored is available for inspection.   In fact, a computer program
can  be  made to  predict how  it  would react  to  particular inputs
provided it has enough free storage to perform the calculation.  This
situation smells of paradox, and there  is one.  Namely, if a program
could predict its own actions in less time than it takes to carry out
the action, it could refuse  to do what it has predicted  for itself.
This only  shows that self-simulation  is necessarily a  slow process,
and this is not surprising.

	However,   present    programs    do    little    interesting
introspection.   This is just  a matter of  the undeveloped  state of
artificial  intelligence; programmers  don't yet know  how to  make a
computer program look at itself in a useful way.

#.  %3Consciousness and self-consciousness%1.  In accordance with the
general  approach  of  this  paper,  a  being  is  considered
self-conscious iff it has certain beliefs about itself. However,
we must remember that beliefs are taken as sentences in our language,
and  by ascribing beliefs  we are  not asserting that  the being uses
that language directly or any other language.

	Here is  a hypothesis  arising  from artificial  intelligence
concerning the relation between language and thought.  Imagine a
person or machine  that represents  information internally in  a huge  network.
Each  node of  the  network has  references  to other  nodes  through
relations.    (If the system has  a  variable  collection of
relations, then the relations have to be represented by nodes, and
we get a symmetrical theory if we suppose that each node is connected
to a  set of pairs of other nodes).  We can imagine this structure to
have a long term part and also extremely temporary parts representing
current  %2thoughts%1.    Naturally,  each  being  has  a
its own network depending on  its own experience. A  thought is then  a
temporary  node  currently  being  referenced  by  the  mechanism  of
consciousness.  Its  meaning is determined by its references to other
nodes which  in turn  refer to  yet other  nodes.   Now consider  the
problem of communicating a thought to another being.

	Its full communication would  involve transmitting the entire
network  that  can  be  reached  from  the  given  node, and this would
ordinarily constitute the entire experience of the being.   More than
that,  it would be  necessary to  also communicate the  programs that
that take action on the basis of encountering certain nodes.  Even if
all this could be transmitted, the recipient would still have to find
equivalents  for  the  information  in  terms  of  its  own  network.
Therefore, thoughts  have to  be translated  into  a public  language
before they can be commuunicated.

	A language  is also a network  of associations
and  programs.  However, certain  of the nodes  in this network (more
accurately a %2family%1 of networks, since no two people speak precisely the same
language) are  associated with words  or set phrases.  Sometimes the
translation from thoughts to sentences is easy,
because large  parts of the  private
networks are taken from the public network, and there is an advantage
in preserving the correspondence.  However, the translation is always
approximate (in sense that still lacks a technical definition),
and some  areas of
experience  are  difficult to  translate  at  all.  Sometimes this is for
intrinsic  reasons, and  sometimes because  particular cultures don't
use language in this area.  (It is my impression that cultures differ
in the  extent to which information about  facial appearance that can
be used for recognition is  verbally transmitted).  According to  this
scheme, the "deep structure"  of a publicly expressible thought  is a
node in the public network.  It is translated into the deep structure
of a sentence as a tree  whose terminal nodes are the nodes to  which
words or set  phrases are attached.  This "deep  structure" then must
be translated into a string in a spoken or written language.

	The need to use language to express thought also applies when
we have to ascribe thoughts to other beings, since  we cannot put the
entire network into a single sentence.

→→→→→→→→→→There is more to come here about what ideas are ←←←←←←←←←←←
→→→→→→→→→→needed for self-consciousness.←←←←←←←←←←←←←←←←←←←←←←←←←←←←←

	#. %3Intentions.%1

	We may say that a machine intends to perform an action when it
believes that it will perform the action and it believes that the
action will further a goal.
However, further analysis may show that no such first order definition
in terms of belief adequately describes intentions.  In this case,
we can try a second order definition based on an axiomatization of
a predicate %2I(a,s)%1 meaning that the machine intends the action
⊗a when it is in state ⊗s.

	#. %3Free will%1

	When we program a computer  to  make
choices  intelligently  after determining its options,
examining their consequences, and deciding which
is  most  favorable or most moral or whatever, we must
program it  to  take  an  attitude  towards  its  freedom  of  choice
essentially  isomorphic  to  that  which a human must take to his own.

	We can define whether a particular action
was free or forced relative to a theory
that ascribes beliefs and within which
beings do what they believe will advance their goals.
In such a theory, action is precipitated by a belief of the form
%2I should do X now%1.  We will say that the action was free if
changing the belief to %2I shouldn't do X now%1 would have resulted
in the action not being performed.
This requires that the theory of belief have sufficient Cartesian
product structure so that changing a single belief is defined, but it
doesn't require defining what the state of the world would be if
a single belief were different.

	This isn't the  whole free will story, because  moralists are
also  concerned with whether praise  or blame may be  attributed to a
choice.   The following  considerations would  seem to  apply to  any
attempt to define  the morality of actions in a  way that would apply
to machines:

&. There is unlikely  to be a simple behavioral  definition.  Instead
there would be  a second order definition criticizing predicates that
ascribe morality to actions.


&. The theory must contain at least one axiom of morality that is not
just a statement of physical fact.  Relative to this axiom, judgments
of actions can be factual.

&. The theory of morality will presuppose a theory of belief in which
statements of the form %2"It believed the action would harm someone"%1
are defined.  The theory must ascribe beliefs about others' welfare and
perhaps about the being's own welfare.

&. It might be necessary to consider the  machine as imbedded in some
kind of society in order to ascribe morality to its actions.

&. No  present machines admit  such a belief  structure, and  no such
structure  may be required  to make  a machine with  arbitrarily high
intelligence in the sense of problem-solving ability.

&. It seems  doubtful that morally  judgable machines or  machines to
which rights might legitimately be ascribed are desirable if and when
it becomes possible to make them.

→→→→→→More mental qualities will be discussed.←←←←←←←←←

.SKIP TO COLUMN 1
.ONCE CENTER
OTHER VIEWS ABOUT MIND

→→→→→This section will be written←←←←←
.SKIP TO COLUMN 1
.PORTION NOTES
.ONCE CENTER
NOTES

.RECEIVE;

.ITEM←NOTE;
	#. Philosophy and artificial intelligence.  These fields overlap
in the following way:  In order to make a computer program behave
intelligently, its designer must build into it a view of the world
in general, apart from what they include about
particular sciences.  (The skeptic who doubts
whether there is anything to say about the world apart from the
particular sciences should try to write a computer program that can
figure out how to get to Timbuktoo, taking into account not only
the facts about travel in general but also facts about what people
and documents  have what information, and what information will
be required at different stages of the trip and when and how it is
to be obtained.  He will rapidly discover that he is lacking a %2science
of common sense%1, i.e. he will be unable to formally express and
build into his program "what everybody knows".  Maybe philosophy
could be defined as an attempted %2science of common sense%1,
or else the %2science of common sense%1 should be a definite part
of philosophy.)

	Artificial intelligence has a another component in which
philosophers have not studied, namely %2heuristics%1.  Heuristics
is  concerned  with:  given  the  facts  and  a  goal,  how should it
investigate the possibilities and decide what to do.
On the other hand, artificial intelligence is not much concerned
with aesthetics and ethics.

	Not all approaches to philosophy lead to results relevant to
the artificial intelligence problem.  On the face of it, a philosophy
that entailed the view that artificial intelligence was impossible
would be unhelpful, but besides that, taking artificial intelligence
seriously suggests some philosophical points of view.  I am not sure
that all I shall list are required for pursuing the AI goal -
some of them may be just my prejudices - but here they are:

		&. The relation between a world view and the world
should be studied by methods akin to metamathematics in which
systems are studied from the outside.  In metamathematics we study
the relation between a mathematical system and its models.  Philosophy
(or perhaps %2metaphilosophy%1) should study the relation between
world structures and systems within them that seek knowledge.
Just as the metamathematician can use any mathematical methods
in this study and distinguishes the methods he uses form those
being studied, so the philosopher should use all his scientific
knowledge in studying philosphical systems from the outside.

	Thus the question %2"How do I know?"%1 is best answered by studying
%2"How does it know"%1, getting the best answer that the current state
of science and philosophy permits, and then seeing how this answer stands
up to doubts about one's own sources of knowledge.

	&. We  regard  %2metaphysics%1 as  the study  of the  general
structure  of   the  world  and  %2epistemology%1  as  studying  what
knowledge of  the world  can be  had by  an  intelligence with  given
opportunities  to observe  and experiment.   We  need to  distinguish
between  what can  be determined  about the  structure of  humans and
machines  by  scientific   research  over  a   period  of  time   and
experimenting with many individuals, and  what can be learned by in a
particular situation with particular opportunities to observe.   From
the AI point of view, the latter is as important
as the  former, and we suppose that  philosophers would also consider
it part of epistemology.  The possibilities of reductionism  are also
different  for  theoretical  and  everyday epistemology.    We  could
imagine that the rules of everyday epistemology could be deduced from
a knowledge of physics and the structure of the  being and the world,
but  we  can't see  how  one could  avoid  using  mental concepts  in
expressing knowledge actually obtained by the senses.

	&.  It  is  now  accepted that the basic concepts of physical
theories are far removed from observation.  The  human  sense  organs
are  many  levels  of  organization  removed  from quantum mechanical
states, and we have learned to accept the complication this causes in
verifying physical theories. Experience in trying to make intelligent
computer programs suggests that the  basic  concepts  of  the  common
sense  world  are  also complex and not always directly accessible to
observation.   In  particular,  the  common  sense  world  is  not  a
construct  from  sense  data,  but sense data play an important role.
When a man or a computer program sees a dog, we will  need  both  the
relation  between  the  observer and the dog and the relation between
the observer and the brown patch in order to construct a good  theory
of the event.

	&.  In spirit this paper is  materialist, but it is logically
compatible  with some  other philosophies.   Thus  cellular automaton
models of the  physical world may be  supplemented by supposing  that
certain  complex  configurations  interact with  additional  automata
called   souls   that  also   interact   with  each   other.     Such
%2interactionist  dualism%1   won't  meet   emotional  or   spiritual
objections to  materialism, but it does provide a logical  niche for any
empirically  argued  belief in  telepathy, communication with the dead
and  other   psychic phenomena.
A person who believed the alleged evidence for such phenomena and
still wanted a scientific explanation could model his beliefs
with auxiliary automata.

.SKIP TO COLUMN 1
.ONCE CENTER
REFERENCES

→→→→→→→→References will be supplied←←←←←←←←
.BEGIN VERBATIM

John McCarthy
Artificial Intelligence Laboratory
Stanford University
Stanford, California 94305
.END